Debugging typically involves the use of a debugger, a tool that allows a software developer to observe run-time behavior of a computer program and locate errors. Some debugging commands such as a stop command allow the programmer to halt execution of a running process at any time, while manual insertion of breakpoints allow the programmer to halt the process when predetermined points in the code have been reached. The debuggee runs free until it hits a break op-code in the instruction stream, at which point the operating system (OS) will halt the debuggee until the debugger continues the debuggee. Thus, when debugging a computer-program, the program is either running (i.e., executing as a process) or halted. Certain debugging commands, such as step-into, step over, and step-out commands, can be initiated only in break mode (i.e. when the debuggee is halted), and allow the programmer to step through the program state, observer, and/or modify content of variables, pointers, and/or the like.
A common method to implement debuggers for Virtual Machines (VMs) involves having a helper thread running in the debuggee process to service debugger requests. Extending this to interop-debugging (i.e. debugging both native and managed code together) introduces the problem that the helper thread may block on some native portion of the program being debugged and cause the whole system to be deadlocked. Specifically, suppose that a native thread T is stopped by the debugger while holding a lock L. The debugger will not resume native thread T until it makes some calls to the helper thread. However, the helper thread may attempt to take lock L. The helper thread would then block on the native thread T. The native thread T is blocked on the debugger. The debugger is blocked on the helper thread. Therefore, the whole system is deadlocked.
One approach to solving this problem is to have the helper thread check if locks are available before taking them, but this requires knowing which locks are going to be taken by the helper thread and to be able to query for them. However, OS Application Program Interfaces (APIs) usually do not publish the internal locks they take and even when they do, the locks may change from version to version, making it difficult to know which locks will be taken. Furthermore, OS APIs usually do not provide a way to query for these locks.
Another approach may be to avoid all OS APIs. However, the OS may have functionality the helper thread needs to perform its job, such as allocating or freeing memory. Also, the helper thread generally calls into other parts of the VM to make queries and these parts of the VM may call OS APIs. Therefore, taking this approach is impractical.